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WHAT IS CLAIMED IS: 



1 . 



An automated system for experimentation comprising: 



an experiment engine operable to define an experiment 
relating to various treatments for a set of content elements, 
the experiment engine operable to conduct the experiment over 
a data network; 

an observation module operable to collect observation 
data relating to user behavior for each treatment; and 

a scripting/scheduling engine operable to coordinate the 
operation of the experiment engine and the observation module. 

2 . The system of Claim 1 wherein the experiment engine 
is operable to identify elements of content which may 
influence user behavior. 

3 . The system of Claim 1 wherein the experiment engine 
is operable to implement statistical sampling procedures to 
deliver over the data network the various treatments to 
respective control groups of users. 

4. The system of Claim 1 wherein the experiment engine 
is operable to store data relating to the set of content 
elements, objective user behavior to be observed, and users 
allocated to receive treatment during the experiment. 

5 . The system of Claim 1 wherein the experiment engine 
comprises an experiment manager object operable to control the 
execution of the experiment . 



6 . The system of Claim 1 wherein the experiment engine 
comprises an experiment manager object operable to specify 
different treatments of content. 
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7 . The system of Claim 1 wherein the experiment engine 
comprises an experiment manager object operable to define and 
implement statistical sampling procedures. 

5 8. The system of Claim 1 wherein the experiment engine 

comprises an experiment manager object operable to record the 
experiment under way and the participating users. 

9. The system of Claim 1 wherein the experiment engine 
10 comprises an experiment data store operable to store 
experiment data. 

^ 10. The system of Claim 1 wherein the data network 

3 comprises the Internet. 

n 

5 15 

2 11. The system of Claim 1 wherein the observation module 
U comprises an observation access object operable to provide 

access to the observation data. 

Li 2 0 12. The system of Claim 1 wherein the experiment engine 

l J is operable to generate a set of experiment rules for 

3 allocating treatments during the experiment. 

13 . The system of Claim 1 further comprising an 

25 interface in communication with the experiment engine, the 
interface operable to allow a user to interact with the 
experiment engine to define the experiment. 

14. The system of Claim 1 further comprising an 
30 allocator module operable to systematically allocate a 

treatment to a user during the experiment . 

15. An automated method for experimentation comprising: 
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defining an experiment relating to various treatments for 
a set of content elements; 

conducting the experiment over a data network; 

collecting over the data network observation data 
relating to user behavior for each treatment; and 

generating at least one script to coordinate defining an 
experiment, conducting the experiment, and collecting 
observation data. 

16. The method of Claim 15 wherein defining an 
experiment comprises : 

identifying desired objectives for user behavior; 

identifying which treatments may influence user behavior 
related to the desired objectives; and 

generating the various treatments using different 
combinations of the content elements. 

17. The method of Claim 16 further comprising: 
defining at least one control variable for the various 

treatments; and 

assigning a respective level for the control variable for 
each treatment . 

18. The method of Claim 15 wherein conducting the 
experiment comprises : 

selecting at least one treatment for delivery to users; 

grouping users into a plurality of segments, each segment 
comprising users with similar behavioral characteristics; and 

specifying a particular segment of users to receive the 
selected treatment . 

19. The method of Claim 15 wherein conducting the 
experiment compr i ses : 




statistically sampling to specify a plurality of control 
groups, each control group comprising at least one user; 

specifying a particular treatment to be delivered to the 
at least one user in each control group; 

receiving identical requests for content from the 
respective at least one user in each of the control groups; 
and 

in response to the identical requests, delivering to the 
at least one user in each control group a different treatment. 

20. The method of Claim 19 further comprising observing 
site-related behavior of users receiving the various 
treatments . 

21. The method of Claim 15 wherein the data network 
comprises the Internet. 

22. The method of Claim 20 wherein collecting 
observation data comprises recording observed behavior in each 
control group during the experiment . 

23 . The method of Claim 15 further comprising 
identifying elements of content which potentially influence 
behavior of users. 

24. An automated system for experimentation comprising: 
a content system operable to store content, the content 

comprising a set of content elements; and 

a communication management system in communication with 
the content system and operable to define an experiment 
relating to various treatments for the set of content 
elements, conduct the experiment over a data network, collect 
over the data network observation data relating to user 
behavior for each treatment, and generate at least one script 
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for coordinating the operation of the content system and the 
communication management system. 

25. The system of Claim 24 wherein the communication 
5 management system is operable to identify content elements 

which may influence user behavior. 

26. The system of Claim 24 wherein the communication 
management system is operable to implement statistical 

10 sampling procedures to deliver over the data network the 
various treatments to respective control groups of users. 

; 27 . The system of Claim 24 wherein the communication 

management system is operable to generate a set of experiment 
15 rules for allocating treatments during the experiment. 

28. The system of Claim 24 wherein the content system is 
operable to allocate treatments to users according to the 
experiment rules . 



29. The system of Claim 24 wherein the communication 
management system is operable to group users into a plurality 
of segments, each segment comprising users with similar 
behavioral characteristics . 

30. The system of Claim 24 wherein said data network 
comprises the Internet. 



31. The system of Claim 24 wherein the communication 
3 0 management system is operable to define at least one control 

variable for the various treatments and to assign a respective 
level for the control variable for each treatment. 
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32. The system of Claim 24 wherein the content system 
and the communication management system are connected to the 
data network. 

33. The system of Claim 24 wherein the content system 
comprises an allocator module operable to support an interface 
with the communication management system. 

34. The system of Claim 24 wherein the communication 
management system comprises an allocator interface object 
operable to support an interface with the content system. 

35. The system of Claim 24 further comprising a content 
provider interface operable to support an interface between 
the communication management system and a manager user. 

36. An automated system for experimentation comprising: 
an experiment engine operable to define an experiment 

relating to various treatments for a set of content elements, 
the experiment engine operable to allocate each treatment to a 
separate control group of users over a data network; 

an observation module operable to collect observation 
data relating to user behavior for each treatment; and 

a scripting/ scheduling engine operable to coordinate the 
operation of the experiment engine and the observation module. 

37. The system of Claim 36 wherein the experiment engine 
is operable to identify elements of content which may 
influence user behavior. 

38. The system of Claim 36 wherein the experiment engine 
is operable to implement statistical sampling procedures to 
deliver over the data network the various treatments to 
respective control groups of users. 
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39. The system of Claim 36 wherein the data network 
comprises the Internet. 

5 40. An automated method for experimentation comprising: 

defining an experiment relating to various treatments for 
a set of content elements; 

allocating each treatment to a separate control group of 
users over a data network; 
10 collecting over the data network observation data 

relating to user behavior for each treatment; and 

generating at least one script to coordinate defining an 
• j f experiment, conducting the experiment, and collecting 

(;ft observation data. 

;:s 15 

^ 41. The method of Claim 40 wherein the data network 

: b? 

comprises the Internet. 

1, 3 

liQ 42 . The method of Claim 40 further comprising 

1%. 2 0 identifying elements of content which potentially influence 

f3 behavior of users . 

43. A method for on-line experimentation comprising: 

defining a plurality of treatments for a set of content 
25 elements; 

providing a website on a data network; 

receiving requests for content from users accessing the 
website on the data network; 

statistically sampling to create at least one control 
30 group of users; 

allocating over the data network a first treatment to 
each user in the control group; 
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allocating over the data network a second treatment to 
each user not in the control group; 

collecting observation data for observed behavior of 
users in the control group and users not in the control group 
5 and 

generating at least one script to coordinate allocating 
the first and second treatments and collecting observation 
data . 
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10 44. The method of Claim 43 further comprising 

identifying elements of content which may influence user 
behavior . 

i;S 45. The method of Claim 43 wherein providing a website 

;;!; 15 comprises providing a web page at which the set of content 
elements is available in the form of any of the treatments. 

1;^ 46. The method of Claim 43 wherein the data network 

j;0 comprises the Internet. 
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